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Executive  Summary 


As  new  technologies  are  implemented  in  manufacturing  areas,  inspection  processes  will  be  affected. 
Often  because  product  quality  has  increased  or  automated  inspection  equipment  has  been  introduced,  the 
inspection  activity  may  be  reduced  or  eliminated.  In  many  cases  a  disciplined  process  is  needed  to 
evaluate  performance  data  against  risk  to  determine  if  this  change  is  warranted.  A  tool  has  been 
developed  using  an  Excel®  spreadsheet  with  an  imbedded  checklist,  which  will  guide  users  through  a 
series  of  decision  points  to  make  a  determination  if  a  change  is  warranted.  The  approach  attempts  to 
answer  the  question:  How  can  inspection  practices  be  identified,  reviewed,  and  updated  to  keep  up  with 
technological  changes,  trend  results,  facility  changes,  etc.?  This  tool  uses  relevant  manufacturing  data  to 
assist  the  users  in  making  practical  decision  for  reducing  inspection. 
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1.  Introduction 


1.1  Background 

It  has  been  fifty-five  years  since  Department  of  Defense  inspection  requirements  were  given  in  MIL  Q  9858 
Quality  Program  Requirements  [1]  and  MIL  I  45208  Inspection  System  Requirements  [2],  During  the 
intervening  years  product  quality  and  complexity  as  well  as  inspection  capabilities  have  increased  primarily 
through  technological  improvements.  Government  and  commercial  customers  as  well  as  contractors  would 
benefit  from  a  process  to  determine  if  the  current  level  of  inspection  is  appropriate  given  the  technological 
advances.  Decreased  inspection,  where  appropriate,  not  only  reduces  handling  risk  to  hardware  but  has  the 
potential  to  increase  efficiency  of  operations.  For  these  reasons  a  task  team  of  government  and  industry 
representatives  was  established  to  identify  a  process  by  which  manufacturing  and  inspection  processes  could 
be  analyzed  to  determine  the  most  appropriate  inspection  methods  and  tools  to  use. 

Industry  representatives  provided  a  list  of  their  “top  1 0”  processes  that  warranted  investigation  for  change.  The 
methods  used  to  generate  these  lists  varied,  but  typically  included  elements  addressing  the  following  issues: 

•  Process  on  a  program  or  manufacturing  critical  path 

•  Results  from  nonconformity  data  analysis 

•  Process  adequately  monitored  and  capable 

•  Measurement  accuracy  required 

•  Redundant  inspections  in  place 

•  Detection  capabilities 

•  Expert  opinion 

•  Severity  of  nonconformities  found  in  inspection 

Lists  provided  from  the  industry  team  members  were  combined,  sorted,  and  evaluated  by  the  team  to  identify 
the  following  twelve  primary  processes  which  should  be  evaluated  for  reduced  inspection: 

•  Electrical  connector  mate/de-mate  (class  II  mates) 

•  Non-flight  and  non-critical  torques 

•  Kit  inspection  prior  to  release  to  factory  floor 

•  Witness  of  test  set-up 

•  Material  verification  by  inspection  of  bonding/staking/potting 

•  Inspection  of  bonding/staking/potting  in  non-hidden  applications 

•  Inspection  of  wire  stripping  and  crimps  on  flight  harnesses 

•  Witness  of  a  critical  move  or  lift 

•  Inspection  of  surface  mounted  components 

•  Fastener  inspection  per  GSFC  requirements 

•  Printed  wiring  assembly  damage  inspection 

•  Paint  or  surface  coating  inspection 
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This  document  presents  a  tool  and  its  associated  guidance  for  applying  a  consistent  method  to  determine  the 
value  of  reducing  inspection  for  any  process.  It  is  particularly  useful  when  a  program,  customer,  or 
manufacturing  operations  group  in  a  company  introduces  a  new  inspection  technology  or  changes  an  existing 
inspection  technology. 

1.2  Application 

The  primary  audience  involves  those  directly  concerned  with  assessing  the  impact  and  benefits  of  using  a  new 
inspection  technology  or  a  modification  of  an  existing  inspection  technology.  The  analyses  required  as  input  to 
the  evaluation  tool  should  reside  in  the  user  organization’s  command  media  and  should  be  performed  by 
personnel  skilled  in  their  use.  If  the  tool  is  used  to  evaluate  a  supplier’s  processes  their  input  should  be 
obtained  to  ensure  they  are  capable  of  supporting  any  changes.  The  tool  is  applicable  to  stakeholder 
requirements  because  it  should  be  used  to  provide  objective  evidence  on  the  impact  on  the  ability  to  verify 
requirements.  The  provided  tool  and  its  output  is  not  intended  to  be  the  final  decision  but  a  basis  for  an 
organization  to  decide  which  of  the  many  potential  modifications  are  most  appropriate  for  its  stakeholders  and 
business  environment. 

1.3  Assumptions 

It  is  assumed  that  the  users  of  this  information  are  adequately  proficient  in  quality,  reliability,  and  other 
associated  disciplines  to  apply  these  tools  effectively.  This  guide  is  not  intended  to  provide  an  overall 
summary  of  disciplines  required  to  make  an  informed  and  appropriately  risk-adjusted  application  of  its 
content.  It  is  left  to  the  implementing  organization  or  user  to  appropriately  address  and  include  customer, 
company,  or  other  stakeholder  input  when  using  this  guide  or  implementing  its  recommendations. 

1.4  Scope 

The  puipose  of  this  guide  is  to  present  a  process  to  determine  if  the  level  of  inspection  is  appropriate  given  that 
product  quality  has  increased  or  automated  inspection  equipment  has  been  introduced.  The  study  presents  a 
spreadsheet  with  an  embedded  checklist  and  a  user’s  guide.  The  intent  is  to  provide  a  structured  process 
approach  for  decision  making  regarding  the  modification  of  inspection  levels. 

The  reasons  that  inspection  processes  may  be  altered  are  many,  but  the  following  are  a  few: 

•  Introduction  of  new  inspection  technologies 

•  Introduction  of  new  manufacturing  technologies 

•  Significant  process  change  or  facility  change 

•  Observed  lack  of  findings  during  historical  inspections 

•  Cost  or  schedule  drivers 

•  Change  in  management  or  customer  requirements 

•  Reduction  in  manufacturing  mishaps 

•  Observed  lack  of  failures 

Each  of  these  changes  requires  a  re-evaluation  of  the  inspection  process  approach.  The  guidance  and  tool 
included  in  this  document  are  intended  to  provide  a  data  and  process  driven  approach  to  making  decisions  on 
quality  deployment  when  processes  change  or  when  external  factors  require  an  alternative  approach  to 
requirement  verification. 
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2.  Inspection  Evaluation  Rationale  and  Tool  Definition 


2.1  Inspection  Process 

Inspection  is  often  considered  a  non-value  added  expense  and  therefore  subject  to  budget  constraints  that  are 
usually  trying  to  reduce  the  inspection  footprint.  Comments  such  as  “inspection  is  a  bottleneck”  or  “inspection 
is  too  expensive”  are  representative  of  this  viewpoint.  If  these  arguments  lead  to  a  reduction  in  inspection 
without  proper  analysis,  the  organization  may  be  faced  with  significant  risk  that  will  materialize  in  the  future. 
The  inadequacies  of  inspection  can  be  traced  to  either  improperly  applying  an  inspection  system  or  the 
problems  with  the  individual  inspection  process  as  shown  in  Figure  1.  Aside  from  inexperience  in  designing  an 
inspection  system,  external  forces  such  as  cost  and  schedule  constraints  or  lack  of  funding  to  implement  proper 
inspection  processes  and  tools  can  increase  risks. 
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Figure  1.  Inspection  system  and  inspector  interaction. 

The  inspection  system  is  the  approach  for  deploying  inspectors  and  the  assignment  of  tasks  to  these  inspectors. 
Depending  on  the  nature  of  the  process  to  be  inspected  and  the  quality  level  of  the  particular  process,  the 
inspection  system  may  choose  to  use  1 00%  inspection,  sampled  inspection,  operator  inspection,  patrol 
inspection  (inspector  moves  from  station  to  station),  automated  inspection,  or  a  combination  of  multiple  types. 
Inspector  capability  is  dependent  upon  training  to  standards,  the  capability  of  the  inspection  tools  in  use,  the 
time  allowed  for  inspection,  and  the  record  keeping  provided. 
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The  inspection  result  is  dependent  upon  the  interaction  of  the  inspection  system  and  the  capability  of  the 
individual  inspector.  When  either  the  inspector  or  the  inspection  system  is  poor,  then  inspection  results  are 
erratic  as  shown  in  Figure  1 .  Only  when  both  are  good,  will  the  inspection  result  be  good.  In  general  the 
evaluation  of  the  inspectors  or  the  inspection  system  involves  an  auditing  approach  which  gathers  the  results  of 
the  auditing  effort  over  time. 

Before  significant  and  potentially  expensive  changes  are  made  to  the  manufacturing  or  inspection  processes, 
the  contractor  should  make  sure  that  the  ineffective  inspection  process  isn’t  simply  the  result  of  a  mismatch 
between  inspection  system  and  inspector  deployment. 

2.2  Tool  Output 

The  level  of  inspection  is  dependent  upon  the  capability  of  the  underlying  manufacturing  process  which  in  turn 
can  be  affected  by  design  tolerances  and  key  characteristics.  In  general  process  variability  should  be 
independent  of  the  people  performing  the  process.  The  process  should  have  business  or  product  requirement 
goals  which  are  met.  Ancillary  controls  should  be  in  place  such  as  adequate  worker  training,  use  of  calibrated 
equipment,  and  appropriate  raw  materials  presented  to  the  process. 

When  the  manufacturing  process  changes,  the  inspection  approach  should  be  re-evaluated.  These  changes 
include  introduction  of  new  manufacturing  technologies,  significant  process  changes  or  changes  in  facilities, 
and  design  changes.  On  the  other  hand,  inspection  techniques  may  improve  because  of  the  introduction  of 
inspection  technologies  or  improved  inspection  techniques.  Finally,  inspection  may  be  affected  by  changes  in 
specifications  and  customer  requirements,  cost  and  schedule  drivers  noted  by  management,  or  defect  analysis 
and  other  anomaly  investigations. 

Each  of  these  conditions  requires  an  appropriate  analysis  of  the  relevant  manufacturing,  inspection,  or 
stakeholder  requirements  changes.  The  tools  presented  here  evaluate  the  return  on  investment  of  an  inspection 
change  vs.  the  investment  to  implement  the  change.  Investment  includes  both  the  effort  to  conduct  the  analyses 
if  they  haven’t  been  performed  as  well  as  the  effort  to  modify  inspection.  As  shown  in  Figure  2,  if  the  return 
on  investment  is  high  and  the  investment  is  low,  then  it  is  a  natural  inspection  change  to  make  (Just  do  it).  On 
the  other  hand  if  the  return  is  low  and  the  investment  is  high,  then  it’s  best  to  avoid  the  change  (Forget  it). 
When  the  return  on  investment  is  low  and  the  investment  is  low,  most  likely  these  are  second  in  line  to  tackle 
and  there  may  be  many  of  these  changes  to  make  (Backlog).  Finally  when  the  return  on  investment  is  high  and 
the  investment  is  also  high,  a  strategy  must  be  chosen  to  determine  which  inspection  approach  to  change 
(Strategic). 

Each  tool  asks  the  reviewer  to  evaluate  the  investment  and  the  return  on  investment  of  an  inspection  change 
using  ten  analyses.  The  tool  then  locates  the  coordinates  of  the  analysis  onto  Figure  2  giving  the  reviewer 
guidance  on  whether  a  change  in  the  inspection  process  is  warranted  commensurate  with  the  effort  required  to 
bring  about  the  change. 

2.3  Flowchart  for  Evaluating  a  Change 

The  process  for  identifying  when  an  inspection  process  is  a  candidate  for  modification  follows  the  process 
flow  indicated  in  Figure  3.  The  process  is  generic  in  nature  and  requires  the  user  to  perform  multiple 
evaluations  incorporating  internal  corporate  processes.  These  evaluations  provide  management  a  risk 
assessment  of  the  change.  Various  steps  of  the  process  (described  below)  will  list  the  studies  required  by  the 
tool  to  be  described  in  Section  4.4  so  that  a  clear  link  between  the  flow  diagram  and  the  tool  is  made. 
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Figure  3.  Process  for  identifying  the  potential  for  reduced  inspection  using  evaluation  gates. 
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# 

Step 

Description 

1.0 

Start 

The  decision  to  review  the  inspection  function  with  the  expectation  of 
reducing  inspection  may  result  from  various  changes  to  the  production  line: 
introduction  of  manufacturing  equipment  which  increased  the  quality  of  the 
product;  introduction  of  new  inspection  equipment,  perhaps  automated;  or 
directives  from  management  or  the  stakeholder  resulting  from  data  analysis 
or  change  in  requirements. 

It  is  assumed  that  before  changes  occur,  the  original  manufacturing  and 
inspection  processes  are  well  understood.  In  other  words,  a  baseline  should 
be  established  before  embarking  on  a  change. 

Seminal  military  documentation  [1]  [2]  suggested  that  inspection  should  be 
placed  at  the  earliest  available  point  in  the  build  process  so  that  defects 
would  be  discovered  well  before  they  were  extremely  costly  to  correct.  This 
document  reinforces  the  same  idea  through  a  cost  analysis. 

The  approach  to  evaluating  the  inspection  process  presented  here  includes 
the  analysis  of  nonconformity  data.  It  should  be  noted  that  attribute  data  is 
after-the-fact  and  therefore  inferior  to  variables  data.  When  monitoring  any 
process,  the  goal  should  be  to  monitor  process  shifts  before  defects  occur. 
Contractors  should  strive  to  determine  the  appropriate  variables  data  to 
detect  small  process  shifts. 

2.0 

Critical  process? 

Critical  processes  are  those  where  failure  or  likelihood  of  failure  would 
seriously  endanger  the  safety  of  personnel  or  alternatively,  produce  product 
that  could  seriously  degrade  the  mission  or  result  in  mission  failure.  Critical 
processes  require  more  study  before  changing  their  inspection  functions. 
Critical  processes  are  often  identified  by  the  contractor  in  conjunction  with 
the  customer. 

2.1 

Conduct 

effectiveness 

study 

For  this  step,  the  process  is  not  a  critical  one.  Less  scrutiny  is  required  in  this 
case  and  so  the  use  of  the  tool  described  in  Section  4.4  is  not  as  important. 
Through  the  use  of  data  and  observation,  the  inspection  function  is  evaluated 
for  effectiveness.  This  evaluation  can  include  a  review  of  inspector 
efficiency,  escapes,  differences  between  inspectors,  and  so  on. 

2.2 

Improve 
inspection  or 
reduce/eliminate 

With  this  non-critical  process,  simple  modifications  to  the  inspection  process 
may  be  undertaken  to  reduce  inspection,  perhaps  through  sampling,  or 
elimination  if  it  is  obvious  that  inspection  is  no  longer  needed. 

2.3 

End 

Having  conducted  this  simple  study,  any  changes  should  be  monitored  with 
data  collection  to  assure  the  proper  decision  was  made. 

3.0 

Process  capable? 

The  assumption  for  this  step  is  that  a  new  manufacturing  process  has  been 
introduced  and  that  this  process  is  a  critical  one.  Before  considering  any 
changes  to  the  inspection  function,  the  process  must  be  demonstrated  to  be 
capable. 
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# 

Step 

Description 

3.1 

Produce  capable 
process 

A  capable  process  is  repeatable  and  produces  the  desired  product.  Before 
considering  the  reduction  of  inspection,  the  process  must  be  capable.  Typical 
activities  pursued  include  conducting  a  PFMEA,  monitoring  the  process  for 
stability,  and  conducting  a  pilot  study.  Furthermore,  it’s  necessary  to 
monitor  the  process  for  changes  in  the  nonconformity  rates  and  any  impact 
to  the  inspection  function.  The  tool  analyses  that  address  this  step  are  one 
through  five.  Once  the  process  is  demonstrated  to  be  capable,  then  the 
process  flow  returns  to  step  3.0. 

4.0 

Inspection 

effective? 

An  inspection  function  is  effective  if  it  is  successful  in  finding 
manufacturing  errors.  Often  this  is  determined  by  reviewing  nonconformity 
and  inspector  escape  data.  If  a  manufacturing  process  has  been  changed  then 
it  should  be  determined  that  the  inspection  function  is  still  effective. 

Changes  in  the  manufacturing  process  may  require  new  inspection  tools  or 
techniques.  Alternatively  the  manufacturing  process  may  not  have  been 
changed,  but  new  technology  has  been  introduced  into  the  inspection 
function.  Again  with  the  introduction  of  inspection  change,  it  is  necessary  to 
ensure  that  inspection  is  still  effective. 

4.1 

Perform  RCCA 

If  it  has  been  determined  that  the  inspection  function  is  not  effective,  then  a 
root  cause  corrective  action  must  be  conducted.  A  review  of  the  inspection 
system  and  the  inspector  capabilities  must  be  conducted  as  discussed  in 
Section  4.1.  Variation  between  inspectors  is  determined  with  a  gage 
repeatability  and  reproducibility  (gage  R&R)  study.  The  tool  analyses  that 
address  this  step  are  six  through  eight. 

4.2 

Use  new 
technology? 

This  step  asks  whether  new  technology  will  be  introduced  to  make 
inspection  more  effective.  If  not,  then  a  more  routine  method  of  inspection 
improvement  is  required.  If  technology  is  to  be  introduced,  then  a  longer 
term  cost,  or  equivalently  risk,  study  must  be  performed  and  evaluated  in 
step  5.0. 

4.3 

Modify 
conditions  to 
create  effective 
inspection 

In  this  step  the  assumption  is  that  no  significant  technology  changes  have 
been  made  to  the  inspection  function.  As  a  result,  better  training,  inspection 
system  improvement,  sharing  of  best  practices,  and  reviews  of  requirements 
may  be  sufficient  to  make  inspection  effective.  As  always,  data  analysis  is 
required  to  ensure  inspection  is  effective.  The  process  flow  then  returns  to 
step  4.0. 

fkO 

Cost  study  p  < 
kl/k2?  [5] 

At  this  stage  it  is  assumed  that  the  manufacturing  process  is  capable  and  that 
inspection  is  effective.  The  Deming  rule  is  explained  in  more  detail  in 
Appendix  B.  Basically  the  probability  of  error  is  compared  to  the  cost  of 
inspecting  a  part  divided  by  the  cost  to  remove,  repair,  reassemble,  and  retest 
the  part  if  it  is  found  later  to  be  defective.  As  can  be  seen  from  the  rule,  if  it 
is  very  expensive  to  fix  hardware  at  a  later  stage  (which  is  often  the  case  for 
space  hardware),  then  the  probability  of  error  should  be  very  small. 

Otherwise  inspection  is  required  because  errors  found  downstream  are  more 
expensive  to  correct  than  the  cost  of  inspection.  This  step  is  addressed  by 
analysis  nine  in  the  tool. 

5.1 

Reduce  inspection 

In  this  case  the  probability  of  error  is  sufficiently  low,  possibly  because  of 

the  introduction  of  technology,  that  inspection  may  be  reduced.  Usually 
when  a  significant  reduction  of  inspection  is  proposed,  the  stakeholder 
should  be  involved  in  the  decision.  This  is  addressed  by  analysis  ten  in  the 
tool. 


#  Step  _ Description 

5.2  Maintain  When  the  cost  or  risk  study  does  not  pass  the  test,  then  an  inspection 

inspection  reduction  is  not  advised.  Plans  should  be  made  to  maintain  the  same  level  of 

inspection. 

5.3  End  Steps  5.1  and  5.2  both  lead  to  this  final  step  which  is  the  conclusion  of  the 

inspection  evaluation. 


2.4  Tool  Description 

The  tool  shown  in  Figure  4  requires  the  user  to  evaluate  each  listed  study  for  a  return  on  investment  (ROl) 
as  well  as  the  investment  to  complete  the  study.  As  mentioned  earlier,  investment  includes  the  effort  to 
conduct  the  study  as  well  as  the  investment  to  change  the  inspection  approach.  Referring  to  study  #1,  for 
impact  the  user  must  estimate  that  when  the  study  is  completed,  on  a  scale  of  inspector  reduction  (100% 
reduction  being  the  highest  ROl),  what  is  the  likely  ROl?  Referring  to  the  first  study  question  in  Figure  4 
(Do  the  results  of  a  PFMEA  show  potential  for  reducing  inspectors?)  the  rating  is  given  as  a  75% 
reduction.  Apparently  the  PFMEA  indicated  that  the  number  of  defects  would  reduce  significantly  and 
that  the  inspection  process  could  be  modified.  However,  the  study  did  not  go  so  far  as  to  say  inspection 
could  be  eliminated. 

Each  of  the  ten  studies  to  be  conducted  are  briefly  discussed  below. 


#  Study _ Discussion 

1.  PFMEA  Do  the  results  of  a  PFMEA  show  potential  for  reducing  inspectors? 

Decision  making  should  be  based  on  risk  priority  numbers  or  an  equivalent 
approach. 

Often  this  analysis  is  either  skipped  or  done  as  an  afterthought.  A  PFMEA 
assigns  a  risk  priority  number  (RPN)  to  each  manufacturing  process  step. 
(RPN  is  explained  in  the  definition  of  PFMEA.)  It  identifies  potential  risks 
in  the  process  to  prevent  future  nonconformities.  Its  value  is  that  it  in  part 
identifies  high  risk  areas  in  the  process  and  quantitatively  measures  the 
increase  or  reduction  of  risks  based  on  deliberate  improvements.  It  should 
be  noted  that  a  contractor  may  have  an  equivalent  investigation  process  that 
identifies  potential  faults  in  a  manufacturing  process  and  sensitivities  in 
inspection.  It  may  be  appropriate  to  substitute  a  contractor’s  approach  for 
PFMEA.  A  PFMEA  should  take  into  account  the  differences  between 
defects  which  affect  the  intended  use  of  the  product  and  nonconformities 
which  may  be  a  simple  non-fulfilment  of  a  requirement.  Alternatively 
expressed,  a  PFMEA,  or  equivalent  tool,  should  take  into  account  the 
severity  of  nonconformities.  This  is  discussed  further  in  Appendix  B. 

Finally  it  is  advisable  to  verify  that  past  issues  requiring  inspection  are  still 
addressed  or  are  no  longer  present  because  of  technology  advances. _ 

2.  Process  Is  the  new  process  qualified  and  capable?  Based  on  what  is  known  about 

qualification  &  the  new  process,  can  inspectors  be  reduced?  Conclusions  should  be  based 
capability  on  data. 

Qualification  is  a  process  specified  by  internal  procedures  as  well  as 
stakeholder  requirements  and  assures  the  process  produces  an  acceptable 
product  or  output.  This  analysis  becomes  important  if  the  manufacturing  or 
inspection  process  is  being  altered  or  upgraded  significantly.  Capability 
_ refers  to  the  ability  of  a  process  to  fulfill  requirements  for  a  product.  [3] _ 


9 


# 

Study 

Discussion 

3. 

Pilot 

Does  the  pilot  or  proof  of  concept  indicate  inspectors  can  be  reduced? 

Prior  to  full  implementation  of  the  manufacturing  process,  a  pilot  should  be 
conducted  to  determine  the  manufacturing  change’s  effect  on  inspection. 

4. 

Nonconformity 

rate 

Will  the  new  process  significantly  reduced  nonconformities  and  thus  reduce 
the  number  of  inspectors  needed? 

If  the  rate  of  nonconformities  is  low  (high  quality)  a  longer  evaluation 
period  may  be  required.  It  is  a  key  indicator  on  whether  or  not  inspection 
may  be  modified.  Nonconformities  may  be  found  by  either  production  or 
inspection.  This  analysis  should  take  into  account  the  differences  between 
defects  and  nonconformities,  possibly  by  weighting  nonconformities  on  the 
basis  of  severity.  If  a  process  has  a  low  nonconformance  level  but  defects 
are  noted  having  critical  severity,  a  case  can  be  made  to  not  reduce 
inspection. 

5. 

Production  rate 

Will  the  new  process  total  output  rate  or  output  variability  affect  the 
number  of  inspectors  needed?  Are  any  inspector  bottlenecks  predicted? 

If  a  change  in  production  alters  the  rate  of  production,  it  could  cause 
inspection  bottlenecks.  If  the  rate  is  increased,  possibly  more  inspection 
resources  will  be  needed  to  maintain  the  required  throughput,  cycle  time, 
etc.  And  if  enough  time  is  not  given  to  inspection  then  escapes  may  occur. 

6. 

Gage  R&R 

Will  the  process  change  inspector  reproducibility  and  repeatability  or 
equipment  precision  to  tolerance  ratio  cause  a  change  in  inspector 
numbers?  (See  GageR&R  definition) 

Often  bypassed,  a  gage  repeatability  and  reproducibility  study  is  essential 
for  determining  the  two  components  of  measurement  error:  repeatability 
and  reproducibility  of  the  gage.  It  verifies  that  the  training  and  performance 
of  inspectors  as  well  as  the  gage  are  both  appropriate  for  the  measurement 
being  made. 

7. 

Lessons  learned 

Are  the  historical  reasons  for  the  inspector  staffing  levels  still  valid?  Do 
lessons  learned  indicate  a  change  in  inspector  levels  is  warranted? 

Usually  inspection  techniques  were  developed  based  on  particular 
nonconformities  that  were  noted  in  the  past.  Does  the  new  manufacturing 
process  eliminate  the  need  to  inspect  for  any  items? 

8. 

Inspection 

escapes 

Will  the  new  process  help  reduce  inspector  escapes  and  alter  the  number  of 
inspectors  required? 

This  may  take  some  time  to  determine  as  an  escape  may  be  found  months 
after  the  product  first  went  through  inspection. 

9. 

Deming’s  cost 
analysis  [5] 

Does  the  cost  or  risk  to  repair  defects  escaped  from  inspection  justify 
reducing  inspectors.  (See  Appendix  B  for  Deming  rule  discussion.) 

Three  quantities  must  be  determined:  p,  the  probability  of  a  nonconformity 
occurring;  kl,  the  cost  to  inspect  one  part;  and  k2,  the  cost  to  fix  a 
nonconformity  when  it  is  discovered  downstream  which  involves 
dismantling,  repair,  reassemble,  and  retest.  Many  contractors  will  say  they 
simply  do  not  have  these  numbers,  but  upon  further  thought  they  may  well 
have  them  or  be  able  to  estimate  them  based  on  similar  processes  for  which 
there  are  numbers.  For  example,  p  is  commonly  reported,  kl  involves 
dividing  the  total  inspection  time  by  the  number  of  inspections.  k2  may  be 
the  most  difficult  quantity  to  obtain,  but  often  the  cost  of  an  equivalent 
failure  investigation  or  root  cause  corrective  action  is  known:  perhaps  not 
precisely,  but  we  may  only  need  approximate  values.  A  contractor  may 
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#  Study  _ Discussion 

have  an  equivalent  method  for  assessing  risk  and  it  is  appropriate  to 
substitute  such  a  process  for  the  Deming  approach.  It  should  be  noted  that 
Deming’s  approach  in  its  simplest  form  does  not  take  into  account  the 
severity  of  nonconformities  which  is  why  a  PFMEA  stressed  in  the  first 
analysis  is  so  important.  An  example  of  this  analysis  is  given  in  Appendix 
B. 


10.  Stakeholder’s  Will  the  stakeholder  allow  the  change?  Take  into  account  any 
reaction  to  stakeholder-mandated  inspections. 

change  The  results  can  range  anywhere  from  no  changes  are  allowed,  to  allowing 

change  with  evidence,  to  full  agreement  with  the  change.  Obviously  it’s 
best  to  determine  the  stakeholder  opinion  before  embarking  on  a  major 
_ process  change  and  that’s  why,  in  part,  the  tools  were  developed. 


11 


Process  Being  Reviewed 

Printed  wiring  board,  paste  application  process 

Scope 

Flight  hardware 

Nature  of  change 

Stencil  approach  replaced  by  solder  paste  printing  machine 

Process  Revision 

Lead  Reviewer 

Date 

Analysis  Revision 

Quadrant 

STRATEGIC 

Conclusion 

Upgrading  stencil  process  improves  quality  and  inspection  could  be  reduced.  Implementation  is  expensive  though. 

If  a  analysis  is  not  to  be  done,  the  suggested  ranks  are  ROI=Do  not  reduce  inspectors  and  l=High  cost/effort. 
Weight  cells  shaded  in  red  indicate  a  change  from  the  pre-set  weight  and  may  indicate  results  could  be  incorrect. 


Change  due  to  introduction  of 
new  manufacturing 
equipment. 

Weight 

(0- 

100%) 

Click  cell  below 
to  see  pulldown 

Return  on  Investment 
(ROI)  Rank 
( Savings ) 

ROI 

Value 

Click  cell  below 
to  see  pulldown 

Investment  or  Effort 

Rank 

(Cost) 

Effort 

Value 

Studies  to  be  conducted 

Justification 

1 

Do  the  results  of  a  PFMEA  show 
potential  for  reducing  inspectors? 
Decision  making  should  be  based  on  risk 
priority  numbers  or  an  equivalent 
approach. 

This  activity  was  completed  in  September  2014. 
Considerable  increase  in  quality  with  less  rework  than 
use  of  stencil.  A  reduced  inspection  staff  is  likely. 

10% 

Reduce 

inspectors  75% 

$ 

$ 

$ 

0.3 

No  cost/effort 

0 

2 

Is  the  new  process  qualified  and 
capable?  Based  on  what  is  known  about 
the  new  process,  can  inspectors  be 
reduced?  Conclusions  should  be  based 

This  effort  is  underway. 

10% 

Reduce 

inspectors  75% 

$ 

$ 

$ 

0.3 

High  cost/effort 

$ 

$ 

$ 

$ 

0.4 

3 

Does  the  pilot  or  proof  of  concept 
indicate  inspectors  can  be  reduced? 

This  effort  is  underway.  Suspect  it  will  suggest  less 
inspection. 

5% 

Reduce 

inspectors  75% 

$ 

$ 

$ 

0.15 

Medium-high 

cost/effort 

$ 

$ 

$ 

0.15 

4 

Will  the  new  process  significantly 
reduced  nonconformities  and  thus 
reduce  the  number  of  inspectors 
needed? 

Manufacturer  data  indicates  significant 
improvement  over  stencil  approach.  We  should 
conduct  studies  on  our  particular  type  of  hardware. 

10% 

Reduce 

inspectors  75% 

$ 

$ 

$ 

0.3 

Low  cost/effort 

$ 

0.1 

5 

Will  the  new  process  total  output  rate  or 
output  variability  affect  the  number  of 
inspectors  needed?  Are  any  inspector 
bottlenecks  predicted? 

Studies  indicate  a  doubling  in  output  rate.  Inspection 
per  unit  may  not  change  significantly.  Will  need  to 
change  product  flow  to  reduce  inspection 
bottlenecks. 

5% 

Do  not  reduce 

inspectors 

0 

Medium-high 

cost/effort 

$ 

$ 

$ 

0.15 

6 

Will  the  process  change  inspector 

reproducibility  and  repeatability  or 
equipment  precision  to  tolerance  ratio 
causing  a  change  in  inspector  numbers? 
(See  GageR&R  definition) 

This  effort  needs  to  be  conducted.  Suspect  inspection 

variability  would  go  down  as  fewer  items  need  to  be 
checked. 

10% 

Reduce 

inspectors  75% 

$ 

$ 

$ 

0.3 

High  cost/effort 

$ 

$ 

$ 

$ 

0.4 

7 

Are  the  historical  reasons  for  the 

inspector  staffing  levels  still  valid?  Do 
lessons  learned  indicate  a  change  in 
inspector  levels  is  warranted? 

This  effort  needs  to  be  conducted  but  historical 

reasons  are  no  longer  applicable  in  most  cases. 

10% 

Remove 

inspectors 

$ 

$ 

$ 

$ 

0.4 

High  cost/effort 

$ 

$ 

$ 

$ 

0.4 

8 

Will  the  new  process  help  reduce 

inspector  escapes  and  alter  the  number 
of  inspectors  required? 

This  needs  to  be  determined. 

10% 

Reduce 

inspectors  75% 

$ 

$ 

$ 

0.3 

High  cost/effort 

$ 

$ 

$ 

$ 

0.4 

9 

Does  the  cost  or  risk  to  repair  defects 
escaped  from  inspection  justify  reducing 
inspectors.  (See  Appendix  B  for  Deming 
rule  discussion.) 

This  needs  to  be  determined.  Suspect  process  will  not 
affect  metric. 

20% 

Reduce 

inspectors  50% 

$ 

$ 

0.4 

Medium 

cost/effort 

$ 

$ 

0.4 

10 

Will  the  stakeholder  allow  the  change? 
Take  into  account  any  stakeholder 
mandated  inspections. 

Held  customer  meeting  June  2014.  Customer  is  on 
board  with  the  change. 

10% 

Reduce 

inspectors  75% 

$ 

$ 

$ 

0.3 

No  cost/effort 

0 

Total  Weighting  Sum 

100% 

ROI  Score 

2.75 

Investment  Score 

2.4 

Weighted  Return 

69% 

Weighted  Investment 

60% 

INVESTMENT 

Weighted  score  falls  in  the  following  quadrant: 

STRATEGIC 


Figure  4.  Tool  example  for  a  manufacturing  process  change. 
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The  tool  is  dependent  upon  the  weights  assigned  to  each  study.  It  was  recognized  that  the  values  could 
change  depending  on  what  was  causing  inspection  to  be  re-evaluated.  The  three  possibilities  include  the 
following: 

•  Manufacturing  process  change 

•  Inspection  process  change 

•  Data  driven  process  change 

The  suggested  weights  are  included  in  Table  1  below. 


Table  1.  Study  Weights  as  a  Function  of  Type  of  Process  Change 


Study  Line  Number 

Manufacturing 
Process  Change 

Inspection  Process 
Change 

Management  or  Stakeholder 
Input  Change 

1 

Manufacturing 

10% 

10% 

10% 

2 

Manufacturing 

10% 

3% 

3% 

3 

Manufacturing 

5% 

3% 

3% 

4 

Manufacturing 

10% 

3% 

3% 

5 

Manufacturing 

5% 

3% 

3% 

6 

Inspection 

10% 

15% 

8% 

7 

Inspection 

10% 

10% 

10% 

8 

Inspection 

10% 

20% 

20% 

9 

Cost  or  Risk 

20% 

30% 

30% 

10 

Stakeholder  Input 

10% 

3% 

10% 

Notice  that  for  the  manufacturing  process  change,  the  manufacturing  study  lines  are  weighted  more 
heavily.  Similarly  for  the  inspection  process  change,  inspection  study  lines  are  more  heavily  weighted. 
Finally  for  management  or  stakeholder  input,  study  lines  9  and  10  are  more  strongly  weighted.  The  tools 
presented  have  these  weights  built  in.  They  may  be  changed  to  fit  a  user’s  particular  need;  however,  the 
spreadsheet  will  warn  the  user  that  the  weights  have  been  changed. 

2.5  Required  Attributes  and  Skill  Sets  When  Applying  this  Tool 

The  skill  sets  required  for  this  tool  are  basic  Excel®  skills  expected  of  any  quality  engineer.  However, 
the  inputs  to  the  tool  require  significant  analysis  and  judgment  and  require  multiple  other  processes  and 
tools  to  be  used  to  generate  them.  Therefore,  it  is  recommended  that  while  quality  engineering  may  own 
the  tool  and  the  implementation  of  its  output,  there  should  be  a  management  or  steering  organization 
driving  the  analyses  required  to  generate  the  inputs  to  the  tool.  This  may  come  from  multiple 
organizations  or  a  single  organization,  but  in  all  cases  this  organization  should  provide  the  subjective 
review  of  analysis  outputs  from  PFMEAs,  risk  analyses,  personnel  reviews,  etc.  that  provide  an  input  to 
the  tool.  The  management  organization  should  also  be  able  to  provide  input  into  the  weighting  of  the 
inputs  and  how  those  weights  reflect  the  completeness,  risk,  and  opportunities  available  within  each 
analysis. 
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3.  Summary  and  Conclusion 


A  cross-industry  team  was  tasked  with  evaluating  the  inspection  process  in  light  of  modem  manufacturing 
methods.  This  team  developed  generic  evaluation  methodologies  for  a  wide  range  of  inspection  processes 
then  compiled  an  Excel®-based  tool  to  aid  in  systematic  evaluation  and  comparison  across  the  industry. 
The  tool  is  tailored  by  modifying  weights  to  certain  suggested  studies  to  three  different  scenarios:  change 
in  the  manufacturing  process,  the  inspection  process,  and  data  driven  decisions.  Each  tool  weights 
specific  analyses  thought  to  be  important  for  each  situation  and  allows  the  user  to  input  the  investment 
and  return  on  investment  required  for  each  analysis.  The  tools  then  produce  scores  which  guide  the  user 
into  assessing  whether  or  not  it  is  of  value  to  institute  the  anticipated  inspection  modification.  In  the  end, 
after  the  analyses  are  completed  and  used  to  determine  acceptability  of  the  inspection  modification,  then 
an  additional  analysis  may  be  required  to  make  the  final  decision. 

Examples  which  illuminate  the  application  of  the  tool  are  provided  in  Appendix  A.  Ultimately  the  tools 
are  aids  only  and  are  not  a  substitute  for  management  experience  and  business  decision  making. 

Instructions  and  insights  for  the  use  of  this  electronic  tool  are  provided  herein  for  use  by  industry.  The 
tool  may  be  launched  from  the  document  and  is  found  in  Appendix  D. 
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4.  Definitions  and  Acronyms 


Audit 

A  systematic,  independent,  and  documented  process  for  obtaining  audit 
evidence  and  evaluating  it  objectively  to  determine  the  extent  to  which  audit 
criteria  are  fulfilled.  [3] 

Capable 

A  process  is  capable  when  it  is  repeatable  and  produces  the  desired  output 
within  the  expected  cost  constraints.  Capability  refers  to  the  ability  of  a  process 
to  fulfill  requirements  for  a  product.  [3]  In  addition,  this  term  as  used  herein 
includes  the  throughput  required  to  prevent  gridlock  at  the  inspection  process. 

Critical  process 

A  process  whose  failure  or  likelihood  of  failure  would  seriously  endanger  the 
safety  of  personnel  or  alternatively,  produce  product  that  could  seriously 
degrade  the  mission  or  result  in  mission  failure. 

Defect 

Non-fulfilment  of  a  requirement  related  to  an  intended  or  specified  use.  [3] 

Escape 

An  inspection  error  as  evidenced  by  an  issue  found  at  a  higher  level  when  it 
should  have  been  caught  at  the  first  opportunity  presented  to  inspection. 

Gage  R&R 

Gage  R&R  is  used  to  determine  the  inherent  precision  of  the  gage 
(repeatability)  and  the  variation  of  measurement  between  operators  using  the 
gage  (reproducibility).  [4]  It  also  provides  a  precision  to  tolerance  ratio  for  the 
inspection  system  which  is  used  to  determine  if  the  system  is  capable  of 
measuring  the  specified  tolerance. 

Inspection 

Evaluation  by  observation  and  judgment  accompanied  as  appropriate  by 
measurement,  testing,  or  gaging  to  assess  the  conformance  of  supplies  and 
services  to  contract  requirements. 

Inspector 

A  person  authorized  to  perform  a  detailed  examination  of  a  product  with  the 
goal  of  determining  if  the  product  meets  established  requirements. 

Investment 

The  labor  and  capital  equipment  needed  to  evaluate  and  potentially  change  an 
inspection  process.  It  can  include  conducting  studies  and  analyses,  purchasing 
capital  equipment,  and  changing  planning  and  processes  to  accommodate  the 
changes. 

Likelihood 

Likelihood  is  the  chance  that  something  might  happen.  Likelihood  can  be 
defined,  determined,  or  measured  objectively  or  subjectively  and  can  be 
expressed  either  qualitatively  or  quantitatively. 

Method 

Practice,  procedure,  method,  and  work  instruction  are  named  in  accordance 
with  the  nomenclature  of  business  command  media  and  share  the  following 
definition:  A  document  that  defines  what  processes  must  be  performed,  what 
products  must  be  produced,  when  and  how  often  it  must  be  done,  and  who  is 
responsible.  It  may  also  include  the  "How  to"  instructions  that  implement  the 
process. 

Nonconformity 

Non-fulfilment  of  a  requirement.  [3] 

PFMEA 

Process  Failure  Modes  and  Effects  Analysis  is  a  structured  approach  to 
preventive  action  that  assigns  a  risk  priority  number  (RPN)  to  each  process  step 
based  anticipated  failures.  The  RPN  is  the  product  of  frequency  of  occurrence, 
severity,  and  detection. 

Process 

A  series  of  logically  related  activities  comprised  of  value  added  tasks  that 
transforms  an  input  into  a  specific  end  result  or  output  (i.e.,  product  or 
information)  required  by  a  stakeholder. 

Process  control 

A  process  is  under  control  if  it  is  repeatable,  stable,  and  operating  at  its 
designed  target  with  normal  variation. 
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Quality  engineer 

i)  A  person  responsible  for  evaluating  overall  process  performance  on  a  regular 
basis  to  provide  confidence  that  processes  will  satisfy  the  relevant  quality 
requirements  and  recommends  process  changes  when  evidence  indicates  poor 
quality,  ii)  A  person  who  objectively  evaluates  products  or  processes.  Quality 
engineers  provide  stakeholder  advocacy  and  ensure  that  the  design, 
development,  and  production  of  all  products  satisfy  stakeholder  requirements 
and  expectations. 

Requirement 

Need  or  expectation  that  is  stated,  generally  implied,  or  obligatory.  [3] 

Return  on 
Investment  (ROI) 

The  earning  power  for  the  resources  invested  to  implement  a  change  ( e.g . 
project  cost)  by  the  organization.  This  term  is  used  herein  to  indicate  the 
savings  incurred  from  changes  made  to  the  inspection  process. 

RCCA 

Root  Cause  Corrective  Action.  Root  cause  is  the  identification  of  the  failure 
from  which  a  chain  of  effects  or  other  failures  originates.  Corrective  action  is 
the  activity  undertaken  to  eliminate  the  cause  of  a  detected  nonconformity. 

RCCA  efforts  are  usually  deployed  on  situations  of  a  serious  nature. 

Surveillance 

Frequent  observation  of  an  area  or  process  with  the  intent  to  gather  data  on  the 
proper  functioning  of  processes. 

Testing 

A  measurement  used  as  a  means  of  determining  if  functional  requirements  of  a 
product  are  met. 

Tool 

Hardware  or  software  that  automates  some  portion  of  product  or  process 
implementation. 

Verification 

Confirmation  through  the  provision  of  objective  evidence  that  the  requirements 
for  a  specific  intended  use  or  application  have  been  fulfilled.  [3] 
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Appendix  A.  Sample  Tool  Applications 


Three  examples  are  given  below  on  the  application  of  the  tool  which  correspond  to  the  three  tabs  on  the 
Excel®  spreadsheet.  The  first  example  addresses  a  change  in  the  manufacturing  process  while  the  second 
involves  a  change  in  the  inspection  process.  The  third  example  involves  a  data  driven  example  and 
demonstrates  how  the  use  of  the  tool  could  prevent  a  costly  event. 

A.1  Application:  Technology  Introduction  Into  Manufacturing  Process 

A.1.1  Process  Weighted  More  Heavily  Than  Inspection 

The  first  application  involves  the  situation  in  which  the  manufacturing  process  undergoes  a  significant 
upgrade.  This  may  be  the  result  of  the  introduction  of  a  new  manufacturing  technology.  Assuming  the  quality 
of  the  product  is  significantly  improved,  the  level  of  inspection  may  be  reduced  or  eliminated.  The  tool  in  this 
case  emphasizes  the  analyses  that  pertain  to  the  capability  and  stability  of  the  new  process.  (40%  of  the 
weights  are  devoted  to  the  manufacturing  process.)  The  inspection  analyses  explore  whether  or  not  the  fidelity 
of  the  inspection  process  must  be  modified  or  if  inspection  can  be  eliminated. 

A.1. 2  Example:  Paste  Printer  Upgrade  to  Surface  Mount  Technology  (SMT)  Process 

The  tool  shown  in  Figure  5  requires  the  user  to  evaluate  each  listed  study  for  a  ROl  as  a  result  of  the 
investment  or  effort  to  complete  the  study.  ROl  is  a  measure  of  the  impact  on  the  inspection  process:  a  high 
ROl  would  mean  the  study  has  made  a  good  case  for  reducing  inspection.  Note  that  the  weights  are  set,  yet 
they  can  be  changed  by  the  user.  However,  if  they  are  changed,  the  tool  will  flag  this  change  as  a  risk  to  the 
correctness  of  the  conclusion. 

For  ROl,  the  user  must  estimate  using  a  pull  down  menu  of  choices  ranging  from  no  reduction  to  full  reduction 
of  inspectors,  what  is  the  ROl  when  the  analysis  is  completed?  Referring  to  the  first  study  question  in  Figure  5 
(Do  the  results  of  a  PFMEA  show  potential  for  improved  quality?)  the  rating  a  75%  reduction  in  inspection. 
Apparently  the  PFMEA  indicated  that  the  number  of  defects  would  reduce  significantly  and  that  the  inspection 
process  could  be  modified.  However,  the  study  did  not  go  so  far  as  to  say  inspection  could  be  eliminated. 

For  this  same  PFMEA  study,  the  investment  to  complete  the  study  is  rated  the  lowest  rank  of  no  cost/effort. 
This  is  because  as  the  justification  column  indicates,  the  study  is  completed  and  no  further  work  is  required. 
Note  that  some  analyses  will  ultimately  require  the  purchase  of  capital  equipment  to  make  a  proper 
assessment.  For  example,  #2  asks  if  the  process  is  qualified.  This  can’t  be  done  until  the  capital  equipment  is 
installed.  However,  the  cost  is  indicated  as  high  cost/effort  and  the  ROl  is  estimated  as  a  75%  reduction  in 
inspection  personnel.  In  this  case  investment  includes  not  only  the  expense  of  conducting  the  analysis  but  also 
of  acquiring  the  equipment.  Until  the  capital  equipment  is  operational,  ROl  will  have  to  be  estimated. 

Reviewer  responses  at  times  may  appear  to  be  contradictory.  Consider  the  responses  to  analyses  #4,  #7,  and 
#8.  The  response  to  #4  indicates  the  process  will  improve  significantly,  but  #7  and  #8  indicate  a  long  study  is 
still  needed.  The  justification  indicates  the  response  to  #4  is  based  on  manufacturer’s  data  and  applies  to  the 
paste  printing  process  which  is  a  manufacturing  change.  However,  #7  and  #8  refer  to  nonconformities  found 
by  inspectors  and  escapes  respectively.  The  effort  to  determine  these  metrics  will  take  time  and  involve  all 
SMT  process  errors,  not  just  paste  printing.  So  for  example,  if  the  paste  were  perfect,  but  the  oven  profile  was 
incorrect,  many  nonconformities  could  still  result. 

The  tool  indicates  that  there  is  a  significant  case  to  be  made  for  modification  to  the  inspection  process  as 
indicated  by  the  high  ROl  score  of  69%.  However,  the  tool  also  indicates  there  is  a  significant  amount  of  work 
and  capital  expense  required  to  bring  about  the  manufacturing  process  changes  as  indicated  by  the  investment 
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score  of  60%.  This  is  understandable  as  the  paste  printing  equipment  is  purchased,  the  process  must  be 
qualified,  a  pilot  conducted,  and  the  process  must  be  monitored  to  determine  if  the  number  of  nonconformities 
is  reduced.  All  of  this  activity  takes  time  and  expense  which  justifies  the  high  score. 

Referring  to  Figure  2,  this  places  the  process  change  in  the  strategic  quadrant  which  means  management  has 
chosen  to  undertake  this  effort  because  of  the  improvement  to  the  manufacturing  process.  Reduction  of 
inspection  is  of  secondary  concern.  One  might  ask,  if  the  quality  of  the  product  has  improved,  why  would 
inspection  not  be  reduced  or  even  eliminated?  The  answer  is  that  while  placement  of  paste  is  important,  there 
are  other  issues  that  can  cause  nonconformities,  such  as  the  use  of  incorrect  parts,  part  placement,  or  even  oven 
zone  temperatures  used  in  the  SMT  process.  Essentially  the  tool  indicates  the  jury  is  still  out  as  to  whether  or 
not  this  manufacturing  change  will  be  sufficient  to  reduce  or  change  the  inspection  approach.  However,  the 
tool  indicates  that  there  is  a  good  possibility  that  the  inspection  time  per  unit  may  be  reduced.  Note  that  the 
fifth  study  indicates  that  the  rate  of  production  may  go  up  which  may  require  more  inspectors  albeit  a  similar 
or  reduced  inspection  time  per  unit. 
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Process  Being  Reviewed 

Printed  wiring  board,  paste  application  process 

Scope 

Flight  hardware 

Nature  of  change 

Stencil  approach  replaced  by  solder  paste  printing  machine 

Process  Revision 

Lead  Reviewer 

Date 

Analysis  Revision 

Quadrant 

STRATEGIC 

Conclusion 

Upgrading  stencil  process  improves  quality  and  inspection  could  be  reduced.  Implementation  is  expensive  though. 

if  a  analysis  is  not  to  be  done,  the  suggested  ranks  are  ROI=Do  not  reduce  inspectors  and  l=High  cost/effort. 
Weight  cells  shaded  in  red  indicate  a  change  from  the  pre-set  weight  and  may  indicate  results  could  be  incorrect. 


Change  due  to  introduction  of 
new  manufacturing 
equipment. 

Weight 

(0- 

100%) 

Click  cell  below 

to  see  pulldown 

Return  on  Investment 
(ROI)  Rank 
(Savings) 

ROI 

Value 

Click  cell  below 
to  see  pulldown 

Investment  or  Effort 

Rank 

(Cost) 

Effort 

Value 

Studies  to  be  conducted 

Justification 

l 

Do  the  results  of  a  PFMEA  show 
potential  for  reducing  inspectors? 
Decision  making  should  be  based  on  risk 
priority  numbers  or  an  equivalent 
approach. 

This  activity  was  completed  in  September  2014. 
Considerable  increase  in  quality  with  less  rework  than 
use  of  stencil.  A  reduced  inspection  staff  is  likely. 

10% 

Reduce 

inspectors  75% 

$ 

$ 

$ 

0.3 

No  cost/effort 

0 

2 

Is  the  new  process  qualified  and 

capable?  Based  on  what  is  known  about 
the  new  process,  can  inspectors  be 
reduced?  Conclusions  should  be  based 

This  effort  is  underway. 

10% 

Reduce 

inspectors  75% 

$ 

$ 

$ 

0.3 

High  cost/effort 

$ 

$ 

$ 

$ 

0.4 

3 

Does  the  pilot  or  proof  of  concept 
indicate  inspectors  can  be  reduced? 

This  effort  is  underway.  Suspect  it  will  suggest  less 
inspection. 

5% 

Reduce 

inspectors  75% 

$ 

$ 

$ 

0.15 

Medium-high 

cost/effort 

$ 

$ 

$ 

0.15 

4 

Will  the  new  process  significantly 
reduced  nonconformities  and  thus 
reduce  the  number  of  inspectors 
needed? 

Manufacturier  data  indicates  significant 
improvement  over  stencil  approach.  We  should 
conduct  studies  on  our  particular  type  of  hardware. 

10% 

Reduce 

inspectors  75% 

$ 

$ 

$ 

0.3 

Low  cost/effort 

$ 

0.1 

5 

Will  the  new  process  total  output  rate  or 
output  variability  affect  the  number  of 
inspectors  needed?  Are  any  inspector 
bottlenecks  predicted? 

Studies  indicate  a  doubling  in  output  rate.  Inspection 
per  unit  may  not  change  significantly.  Will  need  to 
change  product  flow  to  reduce  inspection 
bottlenecks. 

5% 

Do  not  reduce 
inspectors 

0 

Medium-high 

cost/effort 

$ 

$ 

$ 

0.15 

6 

Will  the  process  change  inspector 

reproducibility  and  repeatability  or 
equipment  precision  to  tolerance  ratio 
causing  a  change  in  inspector  numbers? 
(See  GageR8tR  definition) 

This  effort  needs  to  be  conducted.  Suspect  inspection 

variability  would  go  down  as  fewer  items  need  to  be 
checked. 

10% 

Reduce 

inspectors  75% 

$ 

$ 

$ 

0.3 

High  cost/effort 

$ 

$ 

$ 

$ 

0.4 

7 

Are  the  historical  reasons  for  the 
inspector  staffing  levels  still  valid?  Do 
lessons  learned  indicate  a  change  in 
inspector  levels  is  warranted? 

This  effort  needs  to  be  conducted  but  historical 
reasons  are  no  longer  applicable  in  most  cases. 

10% 

Remove 

inspectors 

$ 

$ 

$ 

$ 

0.4 

High  cost/effort 

$ 

$ 

$ 

$ 

0.4 

8 

Will  the  new  process  help  reduce 
inspector  escapes  and  alter  the  number 
of  inspectors  required? 

This  needs  to  be  determined. 

10% 

Reduce 

inspectors  75% 

$ 

$ 

$ 

0.3 

High  cost/effort 

$ 

$ 

$ 

$ 

0.4 

9 

Does  the  cost  or  risk  to  repair  defects 
escaped  from  inspection  justify  reducing 
inspectors.  (See  Appendix  B  for  Deming 
rule  discussion.) 

This  needs  to  be  determined.  Suspect  process  will  not 
affect  metric. 

20% 

Reduce 

inspectors  50% 

$ 

$ 

0.4 

Medium 

cost/effort 

$ 

$ 

0.4 

10 

Will  the  stakeholder  allow  the  change? 
Take  into  account  any  stakeholder 
mandated  inspections. 

Held  customer  meeting  June  2014.  Customer  is  on 
board  with  the  change. 

10% 

Reduce 

inspectors  75% 

$ 

$ 

$ 

0.3 

No  cost/effort 

0 

Total  Weighting  Sum 

100% 

ROI  Score 

2.75 

Investment  Score 

2.4 

Weighted  Return 

69% 

Weighted  Investment 

60% 

Pick  Chart 


INVESTMENT 


Weighted  score  falls  in  the  following  quadrant: 

STRATEGIC 


Figure  5.  Tool  example  for  a  manufacturing  process  change. 
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A.2  Application:  Technology  Introduction  Into  the  Inspection  Process 

A.2.1  Inspection  Weighted  More  Heavily  Than  the  Manufacturing  Process 

The  second  application  assumes  the  manufacturing  process  has  not  significantly  changed,  but  new  inspection 
equipment  may  be  able  to  eliminate  inspection  and  allow  inspection  to  be  performed  by  the  technician.  In  this 
case  it  is  assumed  the  manufacturing  analyses  are  well  understood  and  up-to-date.  The  emphasis  of  the  tool  is 
on  the  inspection  process  in  this  case.  (45%  of  the  weights  are  devoted  to  the  inspection  process.)  Often  when 
inspection  is  replaced,  some  sort  of  technician  self-check  is  implemented. 

A.2. 2  Example:  Torque  Process 

In  this  particular  example  (Figure  6),  production  management  has  decided  to  purchase  wireless  data  transfer 
torque  wrenches  for  use  in  the  integration  and  test  area.  These  wrenches  will  transmit  the  torque  value  to  the 
planning  software  which  will  check  the  torque  value  with  respect  to  the  specification  and  give  a  pass  or  fail 
notice  to  the  operator.  It  is  believed  that  the  quality  inspection  can  be  eliminated  in  favor  of  operator 
inspection.  Training  of  the  operators  will  have  to  take  place  to  assure  quality  standards  are  maintained. 

Notice  that  some  of  the  questions  in  the  tool  have  been  modified  slightly  over  the  previous  example;  however, 
the  tool  maintains  the  same  structure  of  having  the  first  five  analyses  address  manufacturing,  the  next  three 
address  inspection,  and  the  last  two  address  cost  and  stakeholder  input. 

In  this  example  the  weighted  ROI  value  is  89%  while  the  weighted  investment  is  42%.  Because  there  is 
already  a  low  error  rate,  the  reviewer  indicates  that  for  question  #8  it  will  take  some  time  to  gather  sufficient 
data  to  notice  if  escapes  occur.  The  justification  cell  states  that  there  were  no  errors  in  a  sample  size  of  1 000 
torques,  although  possibly  three  escapes  were  discovered  in  unrelated  investigations.  Although  this  torque 
process  is  about  a  four  sigma  process,  one  would  want  to  have  at  least  a  five  sigma  process  which  would 
require  sample  size  of  at  least  ten  times  the  current  sample  size,  or  10,000  torques,  with  no  additional  errors. 
Management  may  choose  to  replace  quality  inspection  with  operator  inspection,  accept  the  risk,  and  not  wait 
until  data  could  demonstrate  the  change  was  appropriate. 
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Process  Being  Reviewed 

Torque  process 

Scope 

Integration  and  test  torque  process 

Nature  of  change 

1  nstall  wireless  data  transfer  torque  wrenches 

Process  Revision 

Lead  Reviewer 

Date 

Analysis  Revision 

Quadrant 

JUST  DO  IT 

Conclusion 

Tool  suggests  the  change  to  replace  inspection  by  technician  self  check  is  warranted. 

If  a  analysis  is  not  to  be  done,  the  suggested  ranks  are  ROI=Do  not  reduce  inspectors  and  l=High  cost/effort. 
Weight  cells  shaded  in  red  indicate  a  change  from  the  pre-set  weight  and  may  indicate  results  could  be  incorrect. 


Change  due  to  introduction  of 
new  inspection  equipment. 

Weight 

(0- 

100%) 

Click  cell  below 
to  see  pulldown 

Return  on  Investment 

(ROI)  Rank 
(Savings) 

ROI 

Value 

Click  cell  below 
to  see  pulldown 

Investment  or  Effort 

Rank 

(Cost) 

Effort 

Value 

Studies  to  be  conducted 

Justification 

l 

Do  the  results  of  a  PFMEA  show 
potential  for  reducing  inspectors? 
Decision  making  should  be  based  on  risk 
priority  numbers  or  an  equivalent 
approach. 

A  PFMEA  doesn't  show  any  new  failures,  yet  would 
allow  automatic  feedback  to  technician. 

10% 

Remove 

inspectors 

$ 

$ 

$ 

$ 

0.4 

No  cost/effort 

0 

2 

Is  the  new  inspection  process  qualified 
and  capable?  Conclusions  should  be 
based  on  data. 

Process  has  been  qualified  for  years. 

3% 

Remove 

inspectors 

$ 

$ 

$ 

$ 

0.12 

No  cost/effort 

0 

3 

Does  the  pilot  or  proof  of  concept 
indicate  inspectors  can  be  reduced? 

Conducted  pilot  indicates  technician  self  inspection  is 
acceptable.  No  inspection  needed  as  long  as  the  data 
are  recorded. 

3% 

Remove 

inspectors 

$ 

$ 

$ 

$ 

0.12 

No  cost/effort 

0 

4 

Are  there  any  changes  in  requirements 
which  could  affect  identification  of 
defects?  Do  these  changes  affect  the 
number  of  inspectors  required? 

There  are  no  changes  in  requirements. 

3% 

Remove 

inspectors 

$ 

$ 

$ 

$ 

0.12 

No  cost/effort 

0 

5 

Will  the  new  inspection  process  total 
output  rate  or  output  variability  affect 
the  number  of  inspectors  needed?  Are 
any  bottlenecks  predicted? 

Having  technicians  perform  self  check  would  speed 
process  up  significantly.  However  if  inspectors  were 
eliminated  no  bottlenecks  would  occur.  However  an 
adequate  number  of  torque  wrenches  would  have  to 
be  purchased. 

3% 

Remove 

inspectors 

$ 

$ 

$ 

$ 

0.12 

Low  cost/effort 

$ 

0.03 

6 

Will  the  process  change  inspector 
reproducibility  and  repeatibility  or 
equipment  precision  to  tolerance  ratio 
causing  a  change  in  inspector  numbers? 
(See  GageR&R  definition) 

This  effort  was  never  done  on  technicians  that  would 
be  using  the  special  torque  wrenches. 

15% 

Reduce 

inspectors  75% 

$ 

$ 

$ 

0.45 

Medium-high 

cost/effort 

$ 

$ 

$ 

0.45 

7 

Are  the  historical  reasons  for  the 
inspector  staffing  levels  still  valid?  Do 
lessons  learned  indicate  a  change  in 
inspector  levels  is  warranted? 

Historical  reasons  include  verifying  technician 
torqued  to  the  correct  value  and  record  the  value 
properly.  These  actions  would  be  verified  by  the  new 
torque  wrenches  and  automated  planning. 

10% 

Remove 

inspectors 

$ 

$ 

$ 

$ 

0.4 

No  cost/effort 

0 

8 

Will  the  new  process  help  reduce 
inspector  escapes  and  alter  the  number 
of  inspectors  required? 

Escapes  are  expected  to  go  to  zero.  However  previous 
data  indicated  zero  errors  in  1000  torques,  although  3 
escapes  were  discovered  in  unrelated  root  cause 
investigations. 

20% 

Remove 

inspectors 

$ 

$ 

$ 

$ 

0.8 

Medium-high 

cost/effort 

$ 

$ 

$ 

0.6 

9 

Does  the  cost  or  risk  to  repair  defects 
escaped  from  inspection  justify  reducing 
inspectors.  (See  Appendix  B  for  Deming 
rule  discussion.) 

Predictions  are  escapes  would  approach  zero.  Some 
inspection  of  the  planning  software  would  be 
required  to  make  sure  the  computer  is  checking  for 
the  correct  values. 

30% 

Reduce 

inspectors  75% 

$ 

$ 

$ 

0.9 

Medium 

cost/effort 

$ 

$ 

0.6 

10 

Will  the  stakeholder  allow  the  change? 
Take  into  account  any  stakeholder 
mandated  inspections. 

Customer  supports  the  introduction  of  the  new  tool 
wrenches. 

3% 

Remove 

inspectors 

$ 

$ 

$ 

$ 

0.12 

No  cost/effort 

0 

Total  Weighting  Sum 

100% 

ROI  Score 

3.55 

Investment  Score 

1.68 

Weighted  Return 

89% 

Weighted  Investment 

42% 

Pick  Chart 


5 


o 

z 

3 

tj 


INVESTMENT 

Weighted  score  falls  in  the  following  quadrant: 

JUST  DO  IT 


Figure  6.  Tool  example  for  inspection  process  change. 
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A.3  Data  Driven  Inspection  Change 
A.3.1  Management  or  Stakeholder  Request 

The  final  tool  tab  is  used  when  a  management  or  stakeholder  change  is  requested  in  response  to  an  analysis  or 
review  of  feedback  from  design  or  process  related  data.  A  change  may  result  from  data  analysis,  such  as  a 
reduction  in  nonconformities.  The  inspection  process  may  need  to  be  re-evaluated  if  it  is  found  to  be  a 
significant  bottleneck.  Or  inspection  may  be  re-evaluated  if  design  or  customer  requirements  change  based  on 
an  analysis  such  as  recurring  use-as-is  dispositions  for  nonconformities.  This  situation  also  includes  the 
evaluation  of  impediments  to  any  changes,  such  as  operator  concerns  that  jobs  may  be  eliminated.  The  tool  in 
this  case  emphasizes  inspection  studies  (weighted  as  38%)  as  well  as  cost  and  stakeholder  studies  (30%  and 
10%  respectively).  It  is  assumed  the  manufacturing  process  has  been  operating  some  time  as  a  qualified 
process  with  a  low  level  of  nonconformities.  But  since  there  is  a  desire  to  change  inspection,  it  is  important  to 
ensure  the  necessary  inspection  studies  have  been  completed  before  a  change  is  made  to  the  production  line. 

A.3. 2  Example:  Reduced  Inspection  for  Mate/Demate  Process  Based  on  Data  Analysis 

The  tool  requires  the  user  to  evaluate  each  study  listed  for  the  ROI  for  reduced  inspection  as  well  as  the 
investment  to  complete  the  study.  Note  that  the  weights  are  set,  yet  can  be  changed  by  the  user.  However,  if 
they  are  changed,  the  tool  will  flag  this  change  as  a  risk  to  the  correctness  of  the  conclusion.  In  this  example 
(Figure  7)  a  management  decision  resulted  in  changes  to  several  weights. 

For  investment,  the  user  must  estimate  the  effort  to  complete  the  study.  Using  a  pull  down  menu  the  user 
chooses  an  investment  rank  ranging  from  no  cost/effort  to  high  cost/effort.  Note  the  user  will  have  to  match 
their  financial  system  to  the  scale.  For  some  contractors,  $300,000  may  not  be  high  cost  while  for  a  supplier, 
$10,000  could  be  viewed  as  high  cost. 

Referring  to  the  first  study  question  in  Figure  8  (Do  the  results  of  a  PFMEA  show  potential  for  reducing 
inspectors?)  the  investment  rank  is  given  as  high  cost.  The  justification  comment  states  the  study  has  not  been 
done  but  it  is  believed  that  the  effort  would  take  about  six  months  to  complete.  Since  there  are  no  plans  to  do  a 
PFMEA,  the  rules  suggest  that  the  ranks  be  indicated  as  no  reduction  in  inspectors  and  high  cost  for  the 
analyses.  (This  is  indicated  in  red  instructions  near  the  top  of  the  tool  matrix.)  Although  the  management  may 
believe  that  it  is  not  important  to  conduct  a  PFMEA,  the  designers  of  the  tool  believe  it  is  a  necessary  step  in 
minimizing  the  risk  of  a  faulty  conclusion. 

Notice  that  in  this  example,  management  decided  to  also  change  the  weights  of  the  manufacturing  analyses. 
Since  the  PFMEA  was  weighted  as  zero,  the  remaining  weights  of  the  four  manufacturing  analyses  were 
changed  so  that  the  total  sum  of  weights  remained  100%.  The  tool  indicates  these  weight  changes  by  shading 
the  weights  for  the  first  five  analyses  in  red. 

The  tool  indicates  that  there  is  a  significant  case  to  be  made  for  modifying  the  inspection  process  as  indicated 
by  the  high  ROI  score  of  86%.  The  investment  score  is  low  at  18%.  Referring  to  Figure  2,  this  places  the 
process  change  in  the  “just  do  it”  quadrant.  These  scores  imply  there  is  little  risk  to  embarking  on  a 
modification  of  the  inspection  process,  possibly  eliminating  it.  However,  the  red  shaded  weights  for  the 
manufacturing  studies  indicate  the  weightings  have  been  changed  and  may  affect  the  accuracy  of  the  tool 
output.  The  change  in  weight  was  the  result  of  management  not  requiring  a  PFMEA.  This  study  would  identify 
any  particular  failure  mechanisms  in  the  process  which  might  be  important  if  inspection  were  eliminated.  This 
becomes  evident  as  shown  in  the  next  section. 
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Process  Being  Reviewed 

Mate  demate 

Scope 

Test  to  flight  harness  DC  mates 

Nature  of  change 

Review  the  current  process  for  possible  schedule  and  cost  savings  through  the  reduction  of  inspection. 

Process  Revision 

Lead  Reviewer 

Date 

Analysis  Revision 

Quadrant 

JUST  DO  IT 

Conclusion 

For  a  low  effort,  significant  ROI  can  be  achieved.  However  there  are  warnings  based  on  changes  in  the  standard  weights  (as  indicated  by  the  red  cells). 

If  a  analysis  is  not  to  be  done,  the  suggested  ranks  are  ROI=Do  not  reduce  inspectors  and  l=High  cost/effort. 
Weight  cells  shaded  in  red  indicate  a  change  from  the  pre-set  weight  and  may  indicate  results  could  be  incorrect. 


Change  due  to  favorable  data 
results. 

Weight 

(0- 

100%) 

Click  cell  below 
to  see  pulldown 

Return  on  Investment 
(ROI)  Rank 
(Savings) 

ROI 

Value 

Click  cell  below 
to  see  pulldown 

Investment  or  Effort 

Rank 

(Cost) 

Effort 

Value 

Studies  to  be  conducted 

Justification 

l 

Do  the  results  of  a  PFMEA  show 
potential  for  reducing  inspectors? 
Decision  making  should  be  based  on  risk 
priority  numbers  or  an  equivalent 
approach. 

Management  will  not  require  a  PFMEA.  Weight  for 
this  is  changed  from  10  to  0%.  Other  weights  of 
— 

1 

Do  not  reduce 
inspectors 

0 

High  cost/effort 

$ 

$ 

$ 

$ 

0 

2 

Is  the  manufacturing  process  qualified 
and  capable?  Based  on  manufacturing 
process  data,  can  the  number  of 
inspectors  be  reduced? 

~~ 

: 

Remove 

inspectors 

$ 

$ 

$ 

$ 

0.24 

No  cost/effort 

0 

3 

Does  the  proof  of  concept  for  the  data 
driven  inspection  change  indicate 
inspectors  can  be  reduced? 

Since  inspectors  don't  find  many  errors,  the  proof  of 

Remove 

inspectors 

$ 

$ 

$ 

$ 

0.24 

No  cost/effort 

0 

4 

Are  there  any  changes  in  requirements 
which  could  affect  identification  of 
defects?  Do  these  changes  affect  the 
number  of  inspectors  required? 

There  are  no  changes  in  requirements.Some  effort 

■ 

Remove 

inspectors 

$ 

$ 

$ 

$ 

0.24 

Low  cost/effort 

$ 

0.06 

5 

Will  the  total  output  rate  or  output 
variability  affect  the  number  of 
inspectors  needed?  Are  any  inspector 
bottlenecks  predicted? 

Elimination  of  inspection  would  speed  up  the  process, 
_ 

• 

Remove 

inspectors 

$ 

$ 

$ 

$ 

0.16 

No  cost/effort 

0 

6 

Do  the  inspector  reproducibility  and 
repeatibility  measurements  or 
equipment  precision  to  tolerance  ratio 
support  a  change  in  inspector  numbers? 
(See  GageR&R  definition) 

This  effort  has  never  been  done  on  inspectors,  let 
alone  on  operators  who  would  act  as  inspectors. 

8% 

Reduce 

inspectors  75% 

$ 

$ 

$ 

0.24 

Medium 

cost/effort 

$ 

$ 

0.16 

7 

Are  the  historical  reasons  for  the 
inspector  staffing  levels  still  valid?  Do 
lessons  learned  indicate  a  change  in 
inspector  levels  is  warranted? 

There  are  no  lessons  learned  that  the  operators 
would  not  already  be  aware  of. 

10% 

Reduce 

inspectors  75% 

$ 

$ 

$ 

0.3 

No  cost/effort 

0 

8 

Will  a  change  in  inspector  numbers 
affect  escapes?  How  does  this  impact 
the  number  of  inspectors  needed? 

Escapes  are  expected  to  be  low.  Data  for  the  last 
year  showed  14  defects  for  47,019  test  mates. 

20% 

Reduce 

inspectors  75% 

$ 

$ 

$ 

0.6 

Medium 

cost/effort 

$ 

$ 

0.4 

9 

Does  the  cost  or  risk  to  repair  defects 
escaped  from  inspection  justify  reducing 
inspectors.  (See  Appendix  B  for  Deming 
rule  discussion.) 

For  test  mates  nonconformities  can  be  caught  at  the 
next  level  with  little  cost  to  repair. 

30% 

Remove 

inspectors 

$ 

$ 

$ 

$ 

1.2 

No  cost/effort 

0 

10 

Will  the  stakeholder  allow  the  change? 
Take  into  account  any  stakeholder 
mandated  inspections. 

Customer  has  expressed  some  concerns  but  given  a 
tentative  go-ahead. 

10% 

Reduce 

inspectors  50% 

$ 

$ 

0.2 

Low  cost/effort 

$ 

0.1 

Total  Weighting  Sum 

100% 

ROI  Score 

3.42 

Investment  Score 

0.72 

Weighted  Return 

86% 

Weighted  Investment 

18% 

Pick  Chart 


INVESTMENT 

Weighted  score  falls  in  the  following  quadrant: 

JUST  DO  IT 


Figure  7.  Tool  example  for  management  directed  change. 
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A.4  Example:  Enlarging  the  Scope  of  Applicability 

The  change  was  made  to  the  inspection  process  described  in  A3.0  with  a  successful  implementation  in  spite  of 
the  weighting  warning.  But  management  was  so  happy  with  the  result  that  they  decided  to  eliminate  inspection 
on  flight  mates  without  applying  the  tool.  Results  were  not  so  good.  A  costly  mistake  resulted  in  the  need  to 
break  a  thermal  vacuum  test.  Since  the  scope  was  expanded  from  test  to  flight,  the  tool  would  have  to  be 
applied  again.  In  this  case  several  changes  in  the  tool  would  result  in  a  different  conclusion.  Most  likely  the 
tool  would  have  looked  similar  to  Figure  8  and  changes  in  ranks  occurred  for  lines  #4,  #9,  and  #10.  The 
affected  ranks  are  indicated  with  a  red  font.  The  weights  were  restored  to  their  original  values. 

Note  that  the  tool  now  indicates  a  46%  weighted  ROl  score  and  a  61%  weighted  investment  score.  This  places 
the  result  in  the  “FORGET  IT”  quadrant.  The  reason  the  ROI  score  changed  so  significantly  is  that  the  impact 
for  reducing  inspection  dropped  because  of  the  PFMEA  and  cost  analyses.  These  analyses  also  increased  the 
investment  metric  as  significant  resources  would  have  to  be  devoted  to  completing  these  analyses.  Finally, 
note  the  entries  for  the  PFMEA  study.  Since  management  decided  not  to  conduct  this  study,  the  user  followed 
the  template  suggestion  of  leaving  the  entries  in  the  ROI  columns  blank  because  there  is  no  study  to  provide  a 
ROI.  In  addition,  the  investment  rank  is  listed  as  high  cost  because  management  believed  the  investment  was 
excessive. 
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Process  Being  Reviewed 

Mate  demate 

Scope 

Flight  to  flight  mates  prior  to  thermal  vac 

Nature  of  change 

Review  the  current  process  for  possible  schedule  and  cost  savings  through  the  reduction  of  inspection. 

Process  Revision 

Lead  Reviewer 

Date 

Analysis  Revision 

Quadrant 

FORGET  IT 

Conclusion 

Because  mates  cannot  be  inspected  before  thermal  vac,  there  is  a  risk  of  high  cost  if  mate  problem  happens  while  under  test. 

If  a  analysis  is  not  to  be  done,  the  suggested  ranks  are  ROI=Do  not  reduce  inspectors  and  l=High  cost/effort. 
Weight  cells  shaded  in  red  indicate  a  change  from  the  pre-set  weight  and  may  indicate  results  could  be  incorrect. 


Change  due  to  favorable  data 
results. 

Weight 

(0- 

100%) 

Click  cell  below 
to  see  pulldown 

Return  on  Investment 

(ROI)  Rank 
(Savings) 

ROI 

Value 

Click  cell  below 
to  see  pulldown 

Investment  or  Effort 

Rank 

(Cost) 

Effort 

Value 

Studies  to  be  conducted 

Justification 

1 

Do  the  results  of  a  PFMEA  show 

potential  for  reducing  inspectors? 
Decision  making  should  be  based  on  risk 
priority  numbers  or  an  equivalent 
approach. 

Management  will  not  require  a  PFMEA.  Weight  for 

this  is  changed  from  10  to  0%.  Other  weights  of 
manufacturing  studies  were  adjusted  accordingly. 

10% 

Do  not  reduce 

inspectors 

0 

High  cost/effort 

$ 

$ 

$ 

$ 

0.4 

2 

Is  the  manufacturing  process  qualified 
and  capable?  Based  on  manufacturing 
process  data,  can  the  number  of 
inspectors  be  reduced? 

Process  has  been  qualified  for  years. 

3% 

Remove 

inspectors 

$ 

$ 

$ 

$ 

0.12 

No  cost/effort 

0 

3 

Does  the  proof  of  concept  for  the  data 
driven  inspection  change  indicate 
inspectors  can  be  reduced? 

Since  inspectors  don't  find  many  errors,  the  proof  of 
concept  is  obvious. 

3% 

Remove 

inspectors 

$ 

$ 

$ 

$ 

0.12 

No  cost/effort 

0 

4 

Are  there  any  changes  in  requirements 
which  could  affect  identification  of 
defects?  Do  these  changes  affect  the 
number  of  inspectors  required? 

There  are  no  changes  in  requirements.Some  effort 
must  be  expended  to  train  the  operators. 

3% 

Remove 

inspectors 

$ 

$ 

$ 

$ 

0.12 

Medium 

cost/effort 

$ 

$ 

0.06 

5 

Will  the  total  output  rate  or  output 
variability  affect  the  number  of 
inspectors  needed?  Are  any  inspector 
bottlenecks  predicted? 

Elimination  of  inspection  would  speed  up  the  process, 
but  no  bottlenecks  are  expected. 

3% 

Remove 

inspectors 

$ 

$ 

$ 

$ 

0.12 

No  cost/effort 

0 

6 

Do  the  inspector  reproducibility  and 
repeatibility  measurements  or 
equipment  precision  to  tolerance  ratio 
support  a  change  in  inspector  numbers? 
(See  GageR&R  definition) 

This  effort  has  never  been  done  on  inspectors,  let 
alone  on  operators  who  would  act  as  inspectors. 

8% 

Reduce 

inspectors  75% 

$ 

$ 

$ 

0.24 

Medium 

cost/effort 

$ 

$ 

0.16 

7 

Are  the  historical  reasons  for  the 
inspector  staffing  levels  still  valid?  Do 
lessons  learned  indicate  a  change  in 
inspector  levels  is  warranted? 

There  are  no  lessons  learned  that  the  operators 
would  not  already  be  aware  of. 

10% 

Reduce 

inspectors  75% 

$ 

$ 

$ 

0.3 

No  cost/effort 

0 

8 

Will  a  change  in  inspector  numbers 
affect  escapes?  How  does  this  impact 
the  number  of  inspectors  needed? 

Escapes  are  expected  to  be  low.  Data  for  the  last 
year  showed  14  defects  for  47,019  test  mates. 

20% 

Reduce 

inspectors  75% 

$ 

$ 

$ 

0.6 

Medium 

cost/effort 

$ 

$ 

0.4 

9 

Does  the  cost  or  risk  to  repair  defects 

escaped  from  inspection  justify  reducing 
inspectors.  (See  Appendix  B  for  Deming 
rule  discussion.) 

For  test  mates  nonconformities  can  be  caught  at  the 

next  level  with  little  cost  to  repair. 

30% 

Do  not  reduce 
inspectors 

0 

High  cost/effort 

$ 

$ 

$ 

$ 

1.2 

10 

Will  the  stakeholder  allow  the  change? 
Take  into  account  any  stakeholder 
mandated  inspections. 

Customer  has  expressed  some  concerns  but  given  a 
tentative  go-ahead. 

10% 

Reduce 

inspectors  50% 

$ 

$ 

0.2 

Medium 

cost/effort 

$ 

$ 

0.2 

Total  Weighting  Sum 

100% 

ROI  Score 

1.82 

Investment  Score 

2.42 

Weighted  Return 

46% 

Weighted  Investment 

61% 

Pick  Chart 


INVESTMENT 

Weighted  score  falls  in  the  following  quadrant: 

FORGET  IT 


Figure  8.  Tool  example  for  modification  of  an  application  scope  change. 
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Appendix  B.  Deming’s  Rule  Explanation 


B.1  The  Deming  Rule 

The  reason  hardware  for  space  involves  so  much  inspection  can  be  easily  explained  by  the  Deming  rule.  To 
understand  the  method  consider  the  following  definitions: 

p  =  average  percentage  of  nonconformities  taken  over  a  period  of  time.  For  example,  a  printed  wiring  board 
facility  estimates  that  they  have  10,000  opportunities  for  error  in  any  month.  If  30  nonconformities  were  noted 
in  a  month,  then  p  =  30/10,000  =  0.3% 

kl  =  cost  to  inspect  one  printed  wiring  board.  For  example,  an  inspector  spends  four  hours  to  review  all  the 
solder  joints,  parts,  and  connectors  on  a  board.  If  the  inspector’s  time  to  the  company  is  $50/hour,  then  the  cost 
to  inspect  the  board  is  $200. 

k2  =  cost  to  remove  the  board  from  its  unit,  repair  it,  reassemble  it,  and  retest  the  unit  if  the  board  failed.  In 
this  example,  estimating  the  cost  is  challenging.  But  let’s  say  if  components  are  not  mounted  on  the  board 
properly,  their  leads  can  fail  in  vibration  testing.  By  the  time  the  board  reaches  vibration  testing  it  most  likely 
has  been  tested  by  itself,  placed  in  a  unit  and  given  a  functional  test,  then  placed  into  a  vibration  test  setup. 
Failure  at  this  level  would  mean  all  of  the  previous  costs  mentioned  would  have  been  wasted  as  well  as  a 
schedule  hit  and  the  need  to  remove,  repair,  reassemble,  and  retest.  Let’s  say  total  cost  for  this  is  $100,000. 

To  apply  Deming’s  rule,  we  simply  ask  the  question  is  p  <  kl/k2?  In  this  case  p=0.3%  and  kl/k2  = 
200/100,000  =  0.2%.  Since  p  >  kl/k2  as  in  this  case,  then  100%  inspection  is  in  order. 

For  space  hardware,  the  cost  of  finding  a  nonconformity  downstream  can  be  very  costly  which  usually  makes 
kl/k2  a  small  number.  According  to  the  rule,  100%  inspection  is  the  appropriate  way  to  proceed.  This  is  one 
reason  why  space  hardware  is  inspected  so  frequently  and  at  all  phases  of  the  build.  A  missed  nonconformity 
can  be  very  costly  to  fix.  Space  hardware  must  be  built  with  low  error  rate  processes.  And  as  hardware  moves 
into  the  integration  phase,  processes  must  be  even  more  controlled  to  offset  the  increase  in  the  complexity  of 
operations. 

B.2  Severity  of  Defects 

The  space  industry  does  not  maintain  a  consistent  definition  set  for  major,  minor,  and  critical  elements  of  a 
system  or  product.  The  definition  of  each  of  these  is  left  up  to  the  business  or  program  to  define  as  they  see  fit. 
The  lack  of  a  standard  definition  set  for  defining  severity  levels  limits  the  ability  for  nonconformities  to  be 
standardized  across  an  array  of  programs  thus  leading  to  confusion  on  when  a  process  is  in  control  or  not.  See 
the  Joint  Aeronautical  Commanders  Group  (JACG)  handbook  for  the  approach  used  by  the  aviation  industry. 
[6] 

Deming’s  rule  was  originally  applied  to  incoming  inspection  instead  of  production.  However,  when  applying 
the  rule  to  the  building  of  hardware,  it’s  advised  to  take  into  account  the  severity  of  nonconformities.  For 
example,  loose  fiber  on  a  printed  wiring  board  is  less  costly  to  correct  than  removal  and  replacement  of  a  ball 
grid  array  package.  In  addition,  distinguishing  between  nonconformities  and  defects  (see  definitions)  requires 
attention  to  severity.  To  account  for  this  difference,  defects  may  be  categorized  as  critical,  major,  minor,  or 
some  similar  scale.  Then  a  weight,  or  demerit,  is  assigned  to  each  category  and  the  total  number  of  demerits 
are  calculated  as  a  replacement  for  the  simple  concept  of  nonconformity  count  given  in  section  B1.0.  [4] 

This  is  why  a  PFMEA  is  so  important  because  it  can  take  into  account  the  severity  of  nonconformities  and 
defects  as  well  as  the  likelihood  of  their  occurrence. 
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Appendix  C.  Frequently  Asked  Questions 


My  management  has  directed  us  not  to  perform  one  of  the  analyses.  How  do  I  fill  in  the  investment  and  R01 
entries  for  this  line  item? 

The  argument  used  is  that  the  cost  of  the  analysis  is  considered  by  management  to  not  be  worth  the  investment. 
This  is  equivalent  to  setting  the  investment  rank  to  high  cost/effort.  Since  the  analysis  has  not  been  done,  there 
is  no  ROl,  therefore  leave  all  entries  in  the  ROl  rank  columns  blank  for  this  line  item.  These  entries  have  the 
effect  of  moving  the  point  on  the  pick  chart  closer  to  the  “Forget  It”  quadrant.  Note  instructions  for  this  issue 
are  given  on  the  template. 

/  don ’t  believe  the  weights  are  appropriate  for  the  process  I ’m  working  on.  What  do  I  do? 

The  tool  allows  the  weights  to  be  changed  as  necessary.  Just  make  sure  the  weights  all  add  up  to  100%.  When 
weights  are  changed,  they  will  be  indicated  by  a  red  background.  This  simply  reminds  the  user  that  a  deviation 
from  the  suggested  weights  has  been  taken.  All  calculations  will  be  accordingly  adjusted  to  the  weights  you 
enter. 

/  want  to  print  a  portion  of  the  spreadsheet  but  I  can ’t  set  the  print  area.  How  do  I  do  that? 

The  tool  is  protected  so  that  cells  that  contain  formulas  do  not  inadvertently  get  overwritten.  To  remove 
protection  choose  Review  and  then  select  Unprotect  Sheet  as  shown  below. 
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Once  protection  has  been  removed,  select  your  print  area  (under  Page  Layout)  and  print.  After  printing  you 
might  want  to  protect  the  sheet  again.  Go  to  Review  and  click  on  Protect  Sheet.  A  pop-up  window  will  appear. 
Make  sure  the  boxes  labeled  Protect  worksheet  and  contents  of  locked  cells  and  Select  unlocked  cells  are 
checked.  Then  click  on  OK. 


The  Pick  Chart  has  moved  from  its  original  location.  What  do  1  do  about  this? 

Usually  this  is  caused  by  comments  in  the  justification  column  being  of  different  lengths.  Simply  adjust  each 
line  item  to  be  three  to  five  lines  each. 


The  point  on  the  pick  chart  doesn  ’t  agree  with  my  intuition  of  where  it  should  be.  What  does  this  mean? 

First  check  your  ranks  to  make  sure  they  are  correct.  Investment  rank  ranges  from  no  cost/effort  to  high 
cost/effort  using  the  pulldown  menus.  Similarly  for  ROl  do  not  reduce  inspections  means  there’s  little  payback 
for  the  effort  expended  to  do  the  analyses,  while  remove  inspectors  means  there’s  a  significant  return  of 
investment.  If  your  entries  appear  to  be  correct,  the  tool  may  require  you  to  assess  the  situation  further.  In 
general  the  weights  have  been  established  to  yield  reasonable  but  conservative  results.  Of  course  if  you  have 
changed  the  weights,  then  review  the  weights  to  make  sure  they  are  reasonable. 
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Our  group  cannot  seem  to  agree  upon  rank  values  or  even  the  weights.  Can  you  help? 

A  common  way  to  resolve  these  issues  is  through  voting.  For  example  let’s  say  you  want  to  determine  the 
value  of  a  rank.  You  might  give  each  person  two  votes.  They  can  place  each  vote  on  separate  ranks  or  they  can 
place  two  votes  on  any  one  rank.  The  rank  with  the  largest  number  of  votes  wins.  There  are  all  sorts  of 
variations  to  the  voting  rules  that  can  be  created.  For  instance  you  might  allow  people  to  explain  why  they 
believe  their  vote  is  correct  (in  an  effort  to  encourage  others),  or  the  voting  might  be  silent  or  anonymous  (to 
make  sure  each  person  contributes  equally).  Find  out  what  method  works  best  for  the  group  and  stick  with  the 
rule. 

I  want  to  remove  an  inspection.  Of  course  there ’s  always  the  possibility  that  a  mistake  will  get  through  that  the 
inspection  would  have  caught  can  get  through.  But  the  likelihood  of  that  occurring  is  very  small.  Can  I 
eliminate  the  inspection? 

Of  course  we  suggest  you  complete  all  the  entries  for  the  tool.  The  line  item  that  most  likely  will  cause  you 
trouble  is  line  9  (Deming’s  p  <  kl/k2  rule).  What  is  the  expected  expense  should  the  defect  be  missed?  Where 
might  it  be  discovered?  Is  it  in  integration  and  test  or  on  the  launch  pad?  Would  inspection  always  catch  this 
error,  or  could  there  be  an  escape?  Y ou,  or  better  yet,  management  in  conjunction  with  the  stakeholder  input 
must  decide  whether  they  are  willing  to  accept  the  risk  given  the  low  probability  of  occurrence.  Remember  the 
conservative  approach  is  to  conduct  1 00%  inspection  if  there  is  a  likelihood  of  an  expensive  effort  to  remove, 
repair,  reassemble,  and  retest. 

I  don ’t  understand,  is  investment  the  effort  used  to  change  inspection  or  the  effort  to  conduct  the  analyses? 

It  can  be  both.  Many  organizations  have  not  done  the  analyses,  so  it  might  be  expensive  (large  investment)  to 
conduct  the  analyses.  Sometimes  reducing  inspection  may  be  as  simple  as  re -assigning  the  inspector.  Other 
times  there  are  other  costs  such  as  modifying  planning,  perhaps  training  production  personnel,  etc.  Flowever, 
the  main  emphasis  of  the  tool  is  to  determine  if  it  is  worth  pursuing  the  change  to  inspection;  and  therefore, 
most  of  the  cost  is  tied  up  in  the  analyses.  If  the  tool  suggests  a  reduction  in  inspection  is  reasonable, 
management  has  the  obligation  to  review  any  addition  costs  prior  to  making  the  change. 

Is  there  a  subset  of  analyses  that  are  more  important  than  others?  I  really  want  to  get  a  quick  estimate  as  to 
whether  or  not  I  should  proceed  with  potentially  reducing  inspection. 

The  weights  are  intended  to  guide  the  user  into  understanding  the  relative  importance  of  certain  analyses. 

Some  users  may  initially  put  in  best  estimates  for  certain  analyses.  For  example  you  might  not  want  to  go  into 
a  full  study  of  whether  or  not  inspector  escapes  will  be  reduced  by  proposed  changes,  so  you  might  use 
engineering  judgement  to  make  your  first  pass  through  the  tool.  Obviously  it  is  important  to  identify  any 
roadblocks  before  completing  all  analyses.  For  example  if  the  stakeholder  simply  will  not  allow  a  change, 
clearly  it  might  be  better  to  complete  initially  only  those  studies  that  will  be  useful  in  convincing  the 
stakeholder  to  change  their  position. 

The  tool  plotted  my  point  right  between  two  quadrants  of  the  pick  chart.  Which  quadrant  do  I  choose? 

The  division  between  quadrants  should  not  be  thought  of  as  a  sharp  line  but  rather  a  fuzzy  band.  When  the 
point  is  plotted  near  the  vertical  50%  line  question  whether  the  investment  is  too  great  to  proceed.  When  the 
point  is  near  the  horizontal  50%  line,  question  whether  the  ROl  is  sufficient.  In  either  case  discussion  and 
engineering  judgement  is  in  order  to  make  a  final  determination  on  whether  or  not  to  proceed. 
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Appendix  D.  Imbedded  Excel®  Tool 


Double  click  on  the  paper  clip  to  use  the  imbedded  Excel®  tool.  Choose  the  appropriate  tab  for  your 
application  (manufacturing  process  change,  inspection  process  change,  or  data  driven  process  change)  and 
complete  the  required  information. 

Double  click  on  the  paper  clip  to  see  the  embedded  Excel  spreadsheet,  ft 


30 


AEROSPACE  REPORT  NO. 
TOR-201 5-02544 


Process  Approach  to  Determining  Quality  Inspection  Deployment 


Approved  Electronically  by: 


Todd  M.  Nygren,  GENERAL 
MANAGER 

SYSTEMS  ENGINEERING 
DIVISION 
ENGINEERING  & 
TECHNOLOGY  GROUP 


Jacqueline  M.  Wyrwitzke, 
PRINC  DIRECTOR 
MISSION  ASSURANCE 
SUBDIVISION 
SYSTEMS  ENGINEERING 
DIVISION 
ENGINEERING  & 
TECHNOLOGY  GROUP 


Jackie  M.  Webb-Larkin, 
SECURITY  SPECIALIST  III 
GOVERNMENT  SECURITY 
SECURITY  OPERATIONS 
OPERATIONS  &  SUPPORT 
GROUP 


Technical  Peer  Review  Performed  by: 


Jacqueline  M.  Wyrwitzke, 
PRINC  DIRECTOR 
MISSION  ASSURANCE 
SUBDIVISION 
SYSTEMS  ENGINEERING 
DIVISION 
ENGINEERING  & 
TECHNOLOGY  GROUP 


Cheryl  L.  Sakaizawa, 
ADMINISTRATIVE  SPEC  III 
MISSION  ASSURANCE 
SUBDIVISION 
SYSTEMS  ENGINEERING 
DIVISION 
ENGINEERING  & 
TECHNOLOGY  GROUP 


Eric  S.  Richter,  ENGRG 
SPCLST  SR 
SYSTEMS  AND 
OPERATIONS  ASSURANCE 
DEPT 

MISSION  ASSURANCE 
SUBDIVISION 
ENGINEERING  & 
TECHNOLOGY  GROUP 


©  The  Aerospace  Corporation,  2015. 

All  trademarks,  service  marks,  and  trade  names  are  the  property  of  their  respective  owners. 
SK0791 


